Information processing apparatus and information processing method

ABSTRACT

The present technique relates to an information processing apparatus, and an information processing method in which data of content is acquired, and a first visual field image corresponding to a visual field of a first user is cut out from a content image based on the data of the content. In addition, visual field information representing a visual field of a second user viewing the content image is acquired. Furthermore, in a display apparatus, the first visual field image is displayed, and the visual field of the second user is displayed based on the visual field information of the second user. The present technique can be applied to, for example, a case where a plurality users views content, such as a spherical image and a free viewpoint image.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/628,300, filed on Jan. 3, 2020, which is aNational Stage Entry of Patent Application No. PCT/JP2018/024763, filedon Jun. 29, 2018, which claims priority benefit of Japanese PatentApplication No. JP 2017-137027, filed in the Japan Patent Office on Jul.13, 2017. Each of the above-referenced applications is herebyincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present technique relates to an information processing apparatus, aninformation processing method, and a program, and particularly, to aninformation processing apparatus, an information processing method, anda program that can improve communication regarding, for example, aplurality of users viewing the same content.

BACKGROUND ART

For example, it is assumed that a head mounted display (hereinafter,abbreviated as HMD) is used to view content, such as a spherical imageincluding an image in a case of viewing the whole circumference from apoint of view. In this case, part of a range of the entire image of thecontent is displayed on the HMD. A user wearing the HMD can move thehead of the user to move the range displayed on the HMD to thereby movea range of an image (scene in the image) that can be viewed by the user,that is, a visual field of the user, in the spherical image (forexample, see PTL 1).

For example, when the user wearing the HMD faces the right side, therange displayed on the HMD is moved to the right side of the currentlydisplayed range in the spherical image. Therefore, the user can see theright side of the current visual field. In addition, when, for example,the user wearing the HMD faces above, the range displayed on the HMD ismoved above the currently displayed range in the spherical image.Therefore, the user can see above the current visual field.

CITATION LIST Patent Literature

[PTL 1]

JP 2005-354644A

SUMMARY Technical Problem

Incidentally, in a case where a plurality of users wearing HMDs viewsthe same content, each user can view an arbitrary range in the entireimage of the content. However, the range viewed by a user (visual fieldof a user) and a range viewed by another user (visual field of anotheruser) may not coincide. Therefore, there is room for improvement interms of, for example, communication regarding the plurality of usersviewing the same content.

The present technique has been made in view of such circumstances, andthe present technique can improve communication regarding a plurality ofuses viewing the same content.

Solution to Problem

An information processing apparatus or a program of the presenttechnique is an information processing apparatus or a program forcausing a computer to function as the information processing apparatus,the information processing apparatus including: a content acquisitionunit configured to acquire data of content; an image cutting unitconfigured to cut out a first visual field image corresponding to avisual field of a first user from a content image based on the data ofthe content; a visual field information acquisition unit configured toacquire visual field information representing a visual field of a seconduser viewing the content image; and a display control unit configured tocontrol a display apparatus to display the first visual field image andconfigured to control the display apparatus to display the visual fieldof the second user based on the visual field information of the seconduser.

An information processing method of the present technique is aninformation processing method including: acquiring data of content;cutting out a first visual field image corresponding to a visual fieldof a first user from a content image based on the data of the content;acquiring visual field information representing a visual field of asecond user viewing the content image; and controlling a displayapparatus to display the first visual field image and controlling thedisplay apparatus to display the visual field of the second user basedon the visual field information of the second user.

In the information processing apparatus, the information processingmethod, and the program of the present technique, the data of thecontent is acquired, and the first visual field image corresponding tothe visual field of the first user is cut out from the content imagebased on the data of the content. In addition, the visual fieldinformation representing the visual field of the second user viewing thecontent image is acquired. Furthermore, the first visual field image isdisplayed on the display apparatus, and the visual field of the seconduser is displayed on the display apparatus based on the visual fieldinformation of the second user.

Advantageous Effect of Invention

According to an aspect of the present technique, communication by aplurality of users viewing the same content can be improved.

Note that the advantageous effect described here may not be limited, andthe advantageous effect may be any of the advantageous effects describedin the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of acontent viewing system according to the present technique.

FIG. 2 is a block diagram illustrating a configuration example of aninformation processing apparatus.

FIG. 3 is a block diagram illustrating a configuration example of adisplay apparatus.

FIG. 4 is a diagram illustrating an example of an entire image.

FIG. 5 is a diagram illustrating an example of display of a visual fieldimage cut out from a content image.

FIG. 6 is an enlarged view of a wide area image superimposed on thevisual field image.

FIG. 7 is a diagram illustrating an example of display in a case wherevisual field direction instruction marks are superimposed and displayedon the visual field image.

FIG. 8 is a diagram illustrating an example of display of the visualfield image in a case where a visual field of a user is narrower than avisual field of another user.

FIG. 9 is a diagram illustrating an example of display of the visualfield image in a case where the visual field of the user is wider thanthe visual field of the other user.

FIG. 10 is a diagram illustrating an example of display in a case wherea visual field direction instruction line is superimposed and displayedon the visual field image.

FIG. 11A, FIG. 11B, and FIG. 11C are diagrams illustrating an example ofdisplay of a tracking image that tracks the visual field of the otheruser.

FIG. 12A and FIG. 12B are diagrams illustrating an example of display ofa tracking image that tracks the visual field of the other user.

FIG. 13 is a diagram illustrating an example of display in a case wherea visual field direction instruction mark corresponding to the userspeaking is highlighted.

FIG. 14 is a flow chart describing a visual field informationpresentation process.

FIG. 15A, FIG. 15B, FIG. 15C, FIG. 15D, and FIG. 15E are diagramsillustrating an example of display of the wide area image switched fromthe visual field image.

FIG. 16 is a flow chart describing an image switching process.

FIG. 17A and FIG. 17B are diagrams illustrating an example in which thevisual field image is moved more than an amount of rotation of the headof the user.

FIG. 18A and FIG. 18B are diagrams illustrating a relationship between amotion of a line of sight of the user and the movement of the visualfield image.

FIG. 19A and FIG. 19B are diagrams illustrating a relationship betweenthe motion of the line of sight of the user and the movement of thevisual field image.

FIG. 20 is a diagram for describing a line-of-sight movement angle.

FIG. 21A, FIG. 21B, and FIG. 21C are diagrams for describing a headrotation angle.

FIG. 22 is a flow chart describing a visual field image movementprocess.

FIG. 23 is a diagram for describing a method of selecting avatars to bedisplayed.

FIG. 24 is a diagram illustrating an example of display of the avatars.

FIG. 25 is a flow chart describing an avatar display process.

FIG. 26 is a block diagram illustrating a configuration example of ageneral-purpose computer.

DESCRIPTION OF EMBODIMENT

Hereinafter, the best mode (hereinafter, referred to as embodiment) forcarrying out the present technique will be described in detail withreference to the drawings. Note that the embodiment will be described inthe following order.

1. Configuration Example of Content Viewing System According to PresentTechnique

2. Presentation of Visual Field of Each User

3. Tracking Image That Tracks Visual Field of Another User

4. Highlighting User Speaking

5. Visual Field Information Presentation Process of User Apparatus 20

6. Switch of Visual Field Image and Wide Area Image

7. Movement of Visual Field Image

8. Selection of Avatar to Be Displayed

9. A Series of Processes Executed by Software

Note that in the present specification, it can be assumed that a systemdenotes a set of a plurality of constituent elements (apparatuses,modules (parts), and the like), and not all of the constituent elementshave to be in the same housing. Therefore, both a plurality ofapparatuses housed in separate housings and connected through a networkand one apparatus including a plurality of modules contained in onehousing can be assumed as systems in the present specification.

<1. Configuration Example of Content Viewing System According to PresentTechnique>

FIG. 1 is a block diagram illustrating a configuration example of acontent viewing system according to the present technique.

A content viewing system 10 enables to figure out visual fields of otherusers in a case where a plurality of users views the same content. Notethat the content includes an image. Obviously, there can be voicecorresponding to the image. However, only the image of the content willbe mentioned below, and the voice of the content will be not bementioned.

The timing of the plurality of users viewing the same content does nothave to be the same time. For example, when a user (referred to as userA) views the content, visual field information representing the visualfield of the user A may be saved. At later timing, the visual field ofthe user A may be presented to another user (referred to as user B)viewing the same content based on the saved visual field information ofthe user A.

The content provided by the content viewing system 10 can be a sphericalimage including an image in a case of viewing the whole circumferencefrom a point of view or a free viewpoint image including an image in acase of viewing the whole circumference while the point of view ismoved.

The spherical image and the free viewpoint image can be any of an imageobtained by photographing a real space (photographed image), an image ofthe real space photographed in real time (image being photographed), animage obtained by using computer graphics to generate a VR (virtualreality) space of a game or the like, an image including a virtualobject superimposed on the real space, and the like.

The content viewing system 10 includes user apparatuses 20-1 to 20-N (Nis a natural number) used by users and a server apparatus 40.Hereinafter, the user apparatuses 20-1 to 20-N will be simply referredto as a user apparatus 20 in a case where the user apparatuses 20-1 to20-N do not have to be individually distinguished.

The user apparatus 20 includes an information processing apparatus 21and a display apparatus 22. The information processing apparatus 21mainly executes a process of cutting out part of an image of the contentto generate a visual field image to be displayed on the displayapparatus 22. The display apparatus 22 mainly displays the visual fieldimage. In addition, the display apparatus 22 displays visual fields ofother users using other user apparatuses 20 to present the visual fieldsof the other users to the user of the user apparatus 20.

The user of the user apparatus 20 views the visual field image displayedon the display apparatus 22 in the image of the content. Therefore, therange of the scene in the visual field image is the visual field of theuser viewing the visual field image. The angle of view of the visualfield image, that is, the visual field size of the visual field of theuser viewing the visual field image (visual field size (field of view)of the visual field provided by the display apparatus 22 to the userviewing the visual field image), varies depending on a display device asthe display apparatus 22 or display software that displays the image.For example, there are a display device in which the angle of view ofthe visual field image is 90 degrees, a display device in which theangle of view of the visual field image is 210 degrees, and the like.

An example of the display apparatus 22 includes a display apparatus wornand used on the head of the user, such as AR (augmented reality) glassesand other HMDs. However, the display apparatus 22 may be a planardisplay device, such as a television receiver, or a display device thatprojects an image, such as a projector.

Note that the information processing apparatus 21 and the displayapparatus 22 included in the user apparatus 20 may be integrated or maybe placed in different housings and separately arranged. The connectionbetween the information processing apparatus 21 and the displayapparatus 22 may be wired connection or may be wireless connection.

The server apparatus 40 is connected to the user apparatus 20 throughthe Internet 31. The server apparatus 40 includes a content distributionunit 41, a visual field information management unit 42, and acommunication management unit 43.

The content distribution unit 41 distributes data of content through theInternet 31 according to requests from the user apparatuses 20. The dataof the content may be distributed to the user apparatuses 20 at the sametiming or at different timing.

The visual field information management unit 42 acquires, from each userapparatus 20, visual field information representing the visual field ofthe user at the time that the content is viewed in each user apparatus20 and manages the visual field information.

The visual field information includes at least one of contentidentification information, elapsed time information, point-of-viewinformation, visual field center information, or visual field sizeinformation.

The content identification information is information for identifyingthe content. The elapsed time information exists in a case where theimage of the content changes with time, and the elapsed time informationis information representing the elapsed time from the top of the content(temporal position where the content is reproduced). The point-of-viewinformation is information representing the position of the point ofview in a content space that is a space (of the scene) in the freeviewpoint image in the case the content is a free viewpoint image.

The visual field center information is information representing a visualfield center (coordinates of the visual field center) that is the centerof the visual field image (range as the visual field of the user) cutout from the image of the content in the user apparatus 20, that is, thecontent image based on the data of the content. The visual field sizeinformation is information representing a visual field size that is thesize of the visual field image cut out from the content image in theuser apparatus 20, that is, the size of the range as the visual field ofthe user viewing the visual field image (size of the visual fieldprovided by the display apparatus 22 to the user viewing the visualfield image). Note that the user can arbitrarily set the visual fieldsize within a range permitted by the display apparatus 22. In addition,the visual field center can be considered as the center of the visualfield of the user viewing the visual field image, and the visual fieldsize can be considered as the size of the visual field of the userviewing the visual field image.

The content identification information can be referenced to specify thecontent viewed by the user. The elapsed time information can bereferenced to specify the temporal position (timing) (seek position) ofthe content viewed by the user in the case where the content specifiedby the content identification information is content that changes withtime. The visual field center information, as well as the visual fieldsize information and the point-of-view information if necessary, can bereferenced to specify the visual field of the user viewing the contentspecified by the content identification information, that is, the visualfield image (range of the visual field image) viewed by the user in thecontent image. Note that to roughly figure out the visual field (fieldof view) of the user, the visual field information not including thevisual field size information can be transmitted and received. Inaddition, the visual field size of each user can be assumed as the samefixed value, and the visual field of each user can be specified from thevisual field center.

The communication management unit 43 manages communication, such asexchange of messages using voice or characters, between the usersviewing the same content.

Note that the function of the server apparatus 40 may be provided to atleast one of the user apparatuses 20, and a plurality of userapparatuses 20 including the user apparatus 20 provided with thefunction of the server apparatus 40 may be connected to each otherthrough an intranet or the like.

FIG. 2 is a block diagram illustrating a configuration example of theinformation processing apparatus 21 included in the user apparatus 20.

The information processing apparatus 21 includes a communication unit51, a content holding unit 52, a visual field image determination unit53, an image cutting unit 54, a visual field information holding unit55, an input unit 58, a trigger detection unit 59, and a display controlunit 60.

The communication unit 51 connects to the server apparatus 40 throughthe Internet 31 and functions as a content acquisition unit thatacquires the data of the content. Furthermore, the communication unit 51acquires image data and the like of user images corresponding to theusers (user images representing the users), such as icons and avatars ofthe users. The data and the like of the content acquired by thecommunication unit 51 from the server apparatus 40 are recorded in thecontent holding unit 52. In addition, the communication unit 51 notifiesthe server apparatus 40 of the visual field information, which issequentially generated by the visual field information holding unit 55,of the user (first user) using the user apparatus 20. Furthermore, thecommunication unit 51 functions as a visual field informationacquisition unit that acquires, from the server apparatus 40, the visualfield information of another user (second user) using another userapparatus 20 to view the content. The communication unit 51 outputs thevisual field information of the other user acquired from the serverapparatus 40 to the visual field information holding unit 55.

The content holding unit 52 holds the data of the content acquired fromthe server apparatus 40. In addition, the content holding unit 52 holdsthe image data and the like of the user image, such as an icon and anavatar for representing each user, to be superimposed and displayed onthe image (visual field image) cut out from the content image based onthe data of the content.

The visual field image determination unit 53 determines the visual fieldcenter based on at least one of an amount of movement of the line ofsight notified from a line-of-sight detection unit 71 (FIG. 3 ) of thedisplay apparatus 22 or an amount of movement of the head notified froma head motion detection unit 72 (FIG. 3 ). Furthermore, the visual fieldimage determination unit 53 determines the visual field image (range ofthe visual field image) of the user to be cut out from the content imagebased on the visual field center and the visual field size (visual fieldsize of the visual field provided (limited) by the display apparatus 22to the user viewing the visual field image). For example, in the casewhere the display apparatus 22 is an HMD, the visual field imagedetermination unit 53 determines the visual field center based on atleast one of the movement of the line of sight of the user or themovement of the head of the user associated with the HMD as the displayapparatus 22 and determines the visual field image of the user based onthe visual field center and the visual field size.

In addition, the visual field image determination unit 53 moves thevisual field image of the user based on the line of sight of the user inresponse to approach of the line of sight of the user to an edge of thevisual field image of the user.

Note that the visual field image determination unit 53 can make an angleof rotation, which is the amount of movement of the visual field imageof the user, larger than an angle of rotation of the head of the userbased on the angle of rotation of the line of sight of the user and theangle of rotation of the head.

In addition, the visual field image determination unit 53 can determinean initial position of the visual field image corresponding to thetiming that the user has substantially started to view the content(timing that the user has started to use the user apparatus 20), thatis, an initial position of the visual field center of the visual fieldimage, based on the visual field information of other users. Forexample, the visual field image determination unit 53 can specify, basedon the visual field information of the other users, a region in thecontent image where the visual fields of equal to or more than apredetermined number of other users are gathered and can determine aposition in the region as the initial position of the visual fieldcenter. Furthermore, the visual field image determination unit 53 candetermine the visual field image to be cut out from the content image,that is, determine, as the visual field image, the image of the rangewith the center at the initial position of the visual field center inthe content image, based on the initial position of the visual fieldcenter. In this case, the user can easily communicate with the otherusers right after the user starts to view the content.

The image cutting unit 54 cuts out (to thereby generate) the visualfield image corresponding to the visual field of the user, that is, thevisual field image determined by the visual field image determinationunit 53, from the content image based on the data of the content. Inaddition, the image cutting unit 54 can acquire the visual fieldinformation of another user from the visual field information holdingunit 55 and cut out, as the visual field image, an image in the visualfield size of the user (visual field size indicated in the visual fieldsize information included in the visual field information of the user)or in the visual field size of the other user from the content imageaccording to the visual field information of the other user. The visualfield image can be displayed on the display apparatus 22 as, forexample, a tracking image that tracks the visual field of the otheruser. According to the tracking image, the user can set the visual fieldof the other user as the visual field of the user and view an imagesimilar to the visual field image viewed by the other user.

The visual field information holding unit 55 sequentially updates andholds the visual field information in the user apparatus 20 and outputsthe visual field information to the communication unit 51 to cause thecommunication unit 51 to notify the server apparatus 40 of the visualfield information. In addition, the visual field information holdingunit 55 holds the visual field information of other users acquired bythe communication unit 51 from the server apparatus 40.

The input unit 58 includes an operation device, such as a remotecontrol, a voice input device, such as a microphone, and an imagingdevice, such as a camera. The input unit 58 inputs a key operation ofthe user using the operation device, inputs a speech of the user usingthe voice input device, or inputs an image obtained by taking a pictureof the user using the imaging device.

The trigger detection unit 59 detects a key operation, a voice command,a gesture, or the like as a trigger of predetermined action from anoperation, a speech, an image, or the like of the user input by theinput unit 58.

The display control unit 60 controls the display to cause the displayapparatus 22 to display the visual field image and the user imagerepresenting the user, such as an icon and an avatar corresponding toeach user. In addition, the display control unit 60 controls the displayto cause the display apparatus 22 to display the visual fields(information representing the visual fields) of the users that need tobe displayed, based on the visual field information of each user.

For example, the display control unit 60 causes the display apparatus 22to display a visual field position instruction image indicating theposition of the visual field of another user based on the visual fieldinformation of the other user. The visual field position instructionimage includes a wide area image 112 (FIG. 5 ) including the visualfield image (first visual field image) of the user of the user apparatus20 and the visual field image (second visual field image) correspondingto the visual field of the other user. Furthermore, the visual fieldposition instruction image includes a symbol image that indicates theposition of the visual field of the other user and that is superimposedon the visual field image of the user of the user apparatus 20. Examplesof the symbol image include a visual field direction instruction mark, avisual field direction instruction line, and the like described later.

Note that the display control unit 60 can control the display apparatus22 to switch the visual field image of the user of the user apparatus 20and the wide area image in response to a predetermined trigger. Thepredetermined trigger includes at least one of the key operation, thevoice command, the motion of the head, or the gesture operation of theuser of the user apparatus 20.

In addition, the display control unit 60 superimposes the user image,such as an avatar, of another user on the visual field image of the userand causes the display apparatus 22 to display the image. In a casewhere there are a plurality of other users, the display control unit 60can superimpose at least one of a plurality of user images correspondingto the plurality of other users on the visual field image of the userand cause the display apparatus 22 to display the image.

For example, the display control unit 60 can set priorities of the otherusers for which the user images, such as avatars, are to be superimposedand displayed on the visual field image or the like in the displayapparatus 22 and can cause the display apparatus 22 to display part orall of the user images of the plurality of other users according to thepriorities.

For example, the display control unit 60 can control (determine thepriorities) whether or not to superimpose each of the plurality of userimages of the plurality of other users on the visual field image of theuser according to the positional relationship between the visual fieldimage of the user and each of the visual fields of the plurality ofother users.

In addition, for example, the display control unit 60 can preferentiallysuperimpose, on the visual field image of the user, the user image ofanother user with the visual field relatively close to the visual fieldimage of the user (visual field provided by the visual field image)among the plurality of other users and can cause the display apparatus22 to display the image.

Furthermore, for example, the display control unit 60 can preferentiallysuperimpose, on the visual field image of the user, part of theplurality of user images of the plurality of other users according to ahistory of communication between the user and the plurality of otherusers and can cause the display apparatus 22 to display the image.

FIG. 3 is a block diagram illustrating a configuration example of thedisplay apparatus 22 included in the user apparatus 20. Particularly,FIG. 3 illustrates a configuration example suitable for a case in whichthe display apparatus 22 is an HMD mounted on the head of the user.

The display apparatus 22 includes the line-of-sight detection unit 71,the head motion detection unit 72, a display unit 73, and a voiceinput/output unit 74.

The line-of-sight detection unit 71 detects the line of sight of theuser. For example, the corneal reflex or other arbitrary methods andtechniques can be used to detect the line of sight. Furthermore, theline-of-sight detection unit 71 detects, as an amount of movement of theline of sight, an angle (line-of-sight movement angle) from the middleof the visual field image (visual field middle) displayed on the displayunit 73 to the line of sight of the user and transmits the line-of-sightmovement angle to the visual field image determination unit 53 of theinformation processing apparatus 21.

The head motion detection unit 72 detects, as an amount of movement ofthe head, a head rotation angle of the user wearing the HMD as thedisplay apparatus 22 and transmits the head rotation angle to the visualfield image determination unit 53 of the information processingapparatus 21. The head rotation angle of the user wearing the HMD as thedisplay apparatus 22 is also an angle of rotation of the HMD.

The display unit 73 displays the visual field image and the like basedon an image signal of the visual field image and the like supplied fromthe display control unit 60.

The voice input/output unit 74 includes, for example, a microphone and aspeaker and is configured to output voice of the content (content voicebased on the data of the content) and input speech of the user. Theinput speech of the user is used for, for example, communication betweenthe users viewing the same content. That is, the input speech of theuser is transmitted to the other user apparatuses 20 and output from thespeakers of the voice input/output units 74 of the other userapparatuses 20.

The HMD as the display apparatus 22 includes the line-of-sight detectionunit 71 and the head motion detection unit 72 as described above, andthe movement of the line of sight of the user and the movement of thehead are detected in the HMD (in association with the HMD).

<2. Presentation of Visual Field of Each User>

FIG. 4 is a diagram illustrating an example of an entire image includingan entire content image developed on a plane.

An entire image 100 is an image obtained by using an equirectangularprojection method to develop, on the plane, the entire content imagethat is a spherical image.

Note that the projection for developing, on the plane, the entirecontent image that is a spherical image may be a projection other thanthe equirectangular projection method. For example, a Mercatorprojection method can be used instead of the equirectangular projectionmethod. Hereinafter, the image developed on the plane by using theequirectangular projection method will be referred to as anequirectangular projection. Similarly, an image developed on the planeby using the Mercator projection method will be referred to as aMercator projection.

In this case, it is assumed that the content is shared (viewed) by threeusers A to C. However, the timing of the users A to C viewing thecontent may not be the same time.

The display apparatus 22 used by the user A displays a visual fieldimage generated by cutting out, from the content image, a visual fieldrange 101A according to the visual field center information and thevisual field size information of the visual field information of theuser A (range in the visual field size indicated in the visual fieldsize information, with the center at the visual field center indicatedin the visual field center information). Similarly, the displayapparatus 22 used by the user B displays a visual field image generatedby cutting out, from the content image, a visual field range 101Baccording to the visual field center information and the visual fieldsize information of the visual field information of the user B. Thedisplay apparatus 22 used by the user C displays a visual field imagegenerated by cutting out, from the content image, a visual field range10CB according to the visual field center information and the visualfield size information of the visual field information of the user C.

The visual field range 101A represents the visual field (range of thevisual field) of the user A. Similarly, the visual field range 101Brepresents the visual field of the user B, and the visual field range101C represents the visual field of the user C.

<Method Using Wide Area Image>

FIG. 5 is a diagram illustrating an example of display of a visual fieldimage 111A that is displayed on the display apparatus 22 used by theuser A and that is generated by cutting out the visual field range 101Afrom the content image.

However, in the case of FIG. 5 , the wide area image 112 is superimposedand displayed at a predetermined position in the visual field image 111A(upper left in the case of FIG. 5 ) according to a predeterminedoperation of the user A.

The wide area image 112 is a type of visual field position instructionimage indicating the positions of the visual fields of the other users(in this case, users B and C) sharing (viewing) the same content inorder to present the visual fields of the other users to the user (inthis case, user A).

The wide area image 112 is generated by using all or part of the entireimage 100. That is, the wide area image 112 is generated by cutting out,from the entire image 100, a range including the visual field image ofthe user (first visual field image) and the visual field images of theother users (second visual field images). Therefore, in a case where thevisual fields of a plurality of users sharing the content are dispersed,the wide area image 112 is generated based on an image obtained bycutting out a large range of the entire image 100 or based on all of theentire image 100. In a case where the visual fields of a plurality ofusers sharing the content are concentrated, the wide area image 112 isgenerated based on an image obtained by cutting out a small range fromthe entire image 100. Note that the wide area image 112 can always begenerated based on all of the entire image 100.

FIG. 6 is an enlarged view of the wide area image 112 superimposed onthe visual field image 111A (FIG. 5 ) displayed on the display apparatus22 used by the user A.

The wide area image 112 displayed on the display apparatus 22 of theuser A (display apparatus 22 used by the user A) includes the visualfield image of the user A and the visual field images of the users B andC as other users sharing the same content with the user A. Therefore,the wide area image 112 includes the visual field range (of the visualfield) of the user A and the visual field ranges of the users B and C.Note that although the wide area image 112 is an equirectangularprojection in FIG. 6 , the wide area image 112 may be any of a Mercatorprojection, an aerial view, a bird's eye view, and a plan view such as atwo-dimensional map.

In FIG. 6 , visual field range display 113A representing the visualfield range of the user A, visual field range display 113B representingthe visual field range of the user B, an icon 114B as a user image ofthe user B, visual field range display 113C representing the visualfield range of the user C, and an icon 114C as a user image of the userC are superimposed and displayed on the wide area image 112.

According to the visual field range display 113B and the icon 114B, thevisual field range 101B (FIG. 4 ) of the user B is presented to the userA. Similarly, according to the visual field range display 113C and theicon 114C, the visual field range 101C (FIG. 4 ) of the user C ispresented to the user A.

Therefore, the wide area image 112 is superimposed and displayed on thevisual field image 111A (FIG. 5 ) displayed on the display apparatus 22used by the user A, and the user A can figure out the visual fields ofthe other users (in this case, users B and C) sharing the content.

This can improve the communication regarding a plurality of usersviewing the same content. That is, in a case where, for example, theusers perform communication, such as exchange of messages using voice orcharacters, regarding the things viewed by the users, a situation suchas miscommunication can be suppressed, and smooth communication betweenthe users can be assisted.

Note that instead of superimposing and displaying the wide area image112 on the visual field image 111A (FIG. 5 ), the wide area image 112may be displayed on a display device other than the display apparatus22, such as, for example, a television receiver.

Next, other methods of presenting the visual field ranges of the otherusers sharing the content to the user will be described.

<Method Using Visual Field Direction Instruction Marks>

FIG. 7 is a diagram illustrating an example of display in a case wherevisual field direction instruction marks 121, which represent thedirections of the locations of the visual fields of other users sharingthe content, are superimposed and displayed as symbol images, whichindicate the positions of the visual fields of the other users, on thevisual field image 111A, which is displayed on the display apparatus 22used by the user.

In the example of display of FIG. 7 , a visual field directioninstruction mark 121B corresponding to the user B and a visual fielddirection instruction mark 121C corresponding to the user C aresuperimposed and displayed on the visual field image 111A correspondingto the user A (visual field image of user A).

The visual field direction instruction mark 121B is obtained bysurrounding the icon 114B (FIG. 6 ) corresponding to the user B by agraphic including an acute projection, and the acute projectionindicates the direction of the visual field of the user B.

Similarly, the visual field direction instruction mark 121C is obtainedby surrounding the icon 114C (FIG. 6 ) corresponding to the user C by agraphic including an acute projection (graphic illustrating a so-calledspeech bubble), and the acute projection indicates the direction of thevisual field of the user C.

The visual field direction instruction marks 121B and 121C superimposedon the visual field image 111A displayed on the display apparatus 22used by the user A allow the user A to figure out the directions of thelocations of the visual fields of the other users sharing the content(in this case, users B and C). The user A can, for example, rotate thehead in the upper left direction indicated by the acute projection ofthe visual field direction instruction mark 121B to move the visualfield of the user (user A) to the visual field of the user B.

Similarly, the user A can, for example, rotate the head in the rightdirection indicated by the acute projection of the visual fielddirection instruction mark 121C to move the visual field of the user(user A) to the visual field of the user C.

FIG. 8 is a diagram illustrating an example of display of the visualfield image in a case where the visual field of the user is narrowerthan the visual field of another user.

As illustrated in FIG. 7 described above, when the user A viewing thevisual field image 111A including the superimposed visual fielddirection instruction mark 121C rotates the head in the right directionindicated by the acute projection of the visual field directioninstruction mark 121C, the visual field image 111A (range cut out as thevisual field image 111A) displayed on the display apparatus 22 moves tothe right side of the content image. As a result, the visual field ofthe user (user A) gradually approaches the visual field of the user C.Accordingly, the acute projection of the visual field directioninstruction mark 121C is gradually reduced. Furthermore, when the visualfield of the user (user A) is included in the visual field of the user Cwider than the visual field of the user (user A) (when the visual fieldimage (displayed on the display apparatus 22) of the user A is includedin the visual field image of the user C), the acute projection of thevisual field direction instruction mark 121C disappears as illustratedin FIG. 8 .

Therefore, the size of the projection of the visual field directioninstruction mark 121C superimposed on the visual field image 111Adisplayed on the display apparatus 22 of the user A allows the user A tofigure out the proximity and the degree of coincidence of the visualfield of the user (user A) and the visual field of the other user (inthis case, user C).

Furthermore, when the projection of the visual field directioninstruction mark 121C disappears, it can be determined that the user(user A) and the other user (in this case, user C) are viewing the samething. Therefore, the user A and the user C can perform communication,such as exchange of messages, in this state, and the situation such asmiscommunication can be suppressed.

FIG. 9 is a diagram illustrating an example of display of the visualfield image in a case where the visual field of the user is wider thanthe visual field of another user.

As in FIG. 7 described above, when the user A viewing the visual fieldimage 111A including the superimposed visual field direction instructionmark 121C rotates the head in the right direction indicated by the acuteprojection of the visual field direction instruction mark 121C, thevisual field of the user (user A) gradually approaches the visual fieldof the user C. Accordingly, the acute projection of the visual fielddirection instruction mark 121C is gradually reduced. Furthermore, whenthe visual field of the user C narrower than the visual field of theuser (user A) is included in the visual field of the user (user A), theacute projection of the visual field direction instruction mark 121Cdisappears as illustrated in FIG. 9 , and visual field range display141C representing the visual field of the user C is superimposed anddisplayed on the visual field image 111A.

Therefore, the size of the projection of the visual field directioninstruction mark 121C superimposed on the visual field image 111Adisplayed on the display apparatus 22 used by the user A and the visualfield range display 141C allow the user A to figure out the proximityand the degree of coincidence (overlap) of the visual field of the user(user A) and the visual field of the other user (in this case, user C).

Furthermore, when the projection of the visual field directioninstruction mark 121C disappears, it can be determined that the user(user A) and the other user (in this case, user C) are viewing the samething. Therefore, the user A and the user C can perform communication,such as exchange of messages, in this state, and the situation such asmiscommunication can be suppressed. Note that the visual field directioninstruction mark 121C and the visual field range display 141C are typesof visual field position instruction image indicating the position ofthe visual field of the user C.

<Method Using Visual Field Direction Instruction Line>

FIG. 10 is a diagram illustrating an example of display in a case wherea visual field direction instruction line 131, which represents thedirection of the location of the visual field of another user sharingthe content, is superimposed and displayed as a symbol image, whichindicates the position of the visual field of the other user, on thevisual field image 111A, which is displayed on the display apparatus 22used by the user.

In the example of display of FIG. 10 , a visual field directioninstruction line 131B is superimposed and displayed on the visual fieldimage 111A corresponding to the visual field of the user A.

In the visual field direction instruction line 131B, one end of thestraight line represents the direction of the location of the visualfield of the user B (in this case, upper left direction). In addition,an icon 132B corresponding to the user B is superimposed and displayedon the visual field image 111A near the visual field directioninstruction line 131B in order to indicate that the visual fielddirection instruction line 131B corresponds to the user B.

The visual field direction instruction line 131B superimposed on thevisual field image 111A displayed on the display apparatus 22 used bythe user A allows the user A to figure out the direction of the locationof the visual field of the other user sharing the content (in this case,user B). The user A can, for example, rotate the head in the upper leftdirection along the visual field direction instruction line 131B to movethe visual field of the user (user A) to the visual field of the user B.

This can suppress the situation, such as miscommunication, in the casewhere the users perform communication, such as exchange of messages, andsmooth communication between the users can be assisted.

<3. Tracking Image that Tracks Visual Field of Another User>

The user apparatus 20 can track the visual field of another user ofanother user apparatus 20 based on the visual field information of theother user apparatus 20 acquired from the server apparatus 40.

FIG. 11A, FIG. 11B, and FIG. 11C are diagrams illustrating an example ofdisplay of a tracking image AC displayed on the display unit 73 of thedisplay apparatus 22 when the user apparatus 20 used by the user Atracks the visual field of the user C in the case where the visual fieldimage (range cut out as the visual field image from the content image)of the user A is larger than the visual field image of the user C sothat the visual field of the user A is wider than the visual field ofthe user C.

The user apparatus 20 used by the user A can generate the tracking imageAC by cutting out, as a visual field image, an image in the visual fieldsize of the user A or the visual field size of the user C with thecenter at the visual field center indicated in the visual fieldinformation of the user C, from the content image based on the visualfield information of the user C.

In a tracking image 151AC illustrated in FIG. 11A, visual field rangedisplay 152C representing the visual field of the user C and an icon153C corresponding to the user C are superimposed and displayed on thevisual field image in the visual field size of the user A.

In the tracking image 151AC illustrated in FIG. 11B, masking display 154is superimposed on a range outside the visual field of the user C in thevisual field image in the visual field size of the user A, and the icon153C corresponding to the user C is superimposed and displayed on therange of the visual field of the user C.

In the tracking image 151AC illustrated in FIG. 11C, the visual fieldimage in the visual field size of the user C is expanded to correspondto the visual field size of the user A, and the icon 153C correspondingto the user C is further superimposed and displayed.

FIG. 12A and FIG. 12B are diagrams illustrating an example of display ofthe tracking image AC displayed on the display unit 73 of the displayapparatus 22 when the user apparatus 20 used by the user A tracks thevisual field of the user C in the case where the visual field image ofthe user A is smaller than the visual field image of the user C so thatthe visual field of the user A is narrower than the visual field of theuser C.

In a tracking image 161AC illustrated in FIG. 12A, an icon 162Ccorresponding to the user C and an arrow 163 indicating that the visualfield of the user C includes the visual field of the user A (indicatingthat the visual field of the user C is wider than the visual field ofthe user A) are superimposed and displayed on the visual field image inthe visual field size of the user A.

In the tracking image 161AC illustrated in FIG. 12B, the icon 162C, thearrow 163, and visual field range display 164 representing the visualfield of the user A are superimposed and displayed on an image obtainedby reducing the visual field image in the visual field size of the userC to a size within the visual field size of the user A.

The user apparatus 20 can switch the mode between a normal mode fordetermining the visual field image of the user displayed on the displayapparatus 22 and a tracking mode for displaying the tracking imagesillustrated in FIG. 11A, FIG. 11B, FIG. 11C, FIG. 12A, and FIG. 12Baccording to the motion of the head or the like of the user. Accordingto the tracking mode, the user can view the scene viewed by anotheruser.

<4. Highlighting User Speaking>

When the visual fields (including the directions of the visual fields)of other users are always displayed on the visual field image 111Acorresponding to the visual field of the user A, it may become difficultfor the user A to see the visual field image 111A in a case where thenumber of other users increases. Therefore, in displaying the visualfields of the other users, for example, the visual field of only theuser transmitting a message in communication between the users may bedisplayed, or the visual field of the user transmitting a message may behighlighted.

FIG. 13 is a diagram illustrating an example of display in a case wherethe user apparatus 20 used by the user A highlights a visual fielddirection instruction mark 171C corresponding to the user C in responseto transmission of a message by the user C.

In the case of FIG. 13 , the visual field direction instruction mark171C highlighted by thickening the contour line is superimposed on thevisual field image 111A corresponding to the visual field of the user A.Furthermore, text display 172 (in the case of FIG. 13 , “LOOK HERE”)corresponding to the speech of the user C is superimposed and displayedbelow the visual field direction instruction mark 171C.

Note that to allow the user A to visually and more easily distinguishthe other user transmitting the message and another user nottransmitting a message, a visual field direction instruction mark 171Bcorresponding to the other user not transmitting the message (in thiscase, user B) may be displayed by changing the contour to a broken lineor by reducing the brightness to make the visual field directioninstruction mark 171B inconspicuous.

<5. Visual Field Information Presentation Process of User Apparatus 20>

Next, a visual field information presentation process for presenting thelocations of the visual fields of the other users to the user whiledisplaying the visual field image corresponding to the visual field ofthe user on the display apparatus 22 of the user apparatus 20 will bedescribed.

FIG. 14 is a flow chart describing the visual field informationpresentation process.

In the information processing apparatus 21 (FIG. 2 ) of the userapparatus 20 used by the user A, the communication unit 51 connects tothe server apparatus 40 through the Internet 31 and acquires the data ofthe content in step S1. The data of the content acquired from the serverapparatus 40 is held in the content holding unit 52.

Furthermore, in step S1, the communication unit 51 starts to acquire thevisual field information of the other users (for example, users B and C)viewing the same content from the visual field information managementunit 42 of the server apparatus 40.

In step S2, the visual field image determination unit 53 sets, as aninitial position of the visual field center, a position in a rangeviewed by more other users based on the visual field information of theother users. Furthermore, the visual field image determination unit 53determines the visual field image (corresponding to the visual field) ofthe user in the content image based on the initial position of thevisual field center and the visual field size. By setting the initialposition of the visual field center at the position in the range viewedby more other users, the user (in this case, user A) can immediatelyview, as the visual field image, the same image as the other users, andthe users can easily communicate.

In step S3, the image cutting unit 54 cuts out the visual field image111A (FIG. 5 ) determined by the visual field image determination unit53 from the content image to generate the visual field image 111A.

In step S4, the display control unit 60 superimposes, on the visualfield image 111A, visual field position instruction images as displayrepresenting the visual fields of the other users. Subsequently, thedisplay control unit 60 supplies an image signal of the visual fieldimage 111A including the superimposed visual field position instructionimages to the display apparatus 22 and causes the display unit 73 todisplay the visual field image 111A.

Here, the visual field position instruction images include the wide areaimage 112 (FIG. 6 ), symbol images, and the like. The symbol imagesinclude the visual field direction instruction marks 121B and 121C (FIG.7 ), the visual field direction instruction line 131B (FIG. 10 ), andthe like.

In step S5, the visual field image determination unit 53 determineswhether or not to move the visual field image displayed on the displayunit 73 based on the amount of movement (angle of rotation) of the lineof sight and the amount of movement (angle of rotation) of the head.Here, in a case where the visual field image determination unit 53determines to move the visual field image, the process proceeds to stepS6.

In step S6, the visual field information holding unit 55 updates andholds the visual field information of the user based on the amount ofmovement of the line of sight and the amount of movement of the head andoutputs the visual field information to the communication unit 51. Thecommunication unit 51 notifies the updated visual field information ofthe user to the server apparatus 40 through the Internet 31. Note thatthe visual field information notified to the server apparatus 40 is tobe supplied to the other user apparatuses 20.

The visual field image determination unit 53 determines the visual fieldimage corresponding to the visual field of the user in the content imagebased on the updated visual field information. The process returns tostep S3, and the subsequent process is repeated until the end of thereproduction of the content. In this way, the visual field imagecorresponding to the visual field of the user in the content image isdetermined based on the updated visual field information, and the visualfield image displayed on the display unit 73 moves based on the amountof movement of the line of sight of the user and the amount of movementof the head (the range cut out as the visual field image from thecontent image moves).

Furthermore, in a case where the visual field image determination unit53 determines not to move the visual field image in step S5, the visualfield image determination unit 53 sets again the visual field image ofthe last time as the visual field image corresponding to the visualfield of the user, and step S6 is skipped. The process then returns tostep S3, and the subsequent process is repeated until the end of thereproduction of the content.

According to the visual field information presentation process describedabove, the visual fields (including the directions of the locations ofthe visual fields) of the other users are displayed to present thevisual fields of the other users to the user. The user can move the lineof sight and the head to move the visual field of the user based on thepresentation, and the user can immediately see the same images as theother users. This can suppress the situation, such as miscommunication,in the communication between the users.

<6. Switch of Visual Field Image and Wide Area Image>

In the description so far, the wide area image (FIG. 6 ) is superimposedand displayed on the visual field image (FIG. 5 ) or displayed onanother display device. However, the display apparatus 22 can switch anddisplay the visual field image and the wide area image.

In switching the display of the display apparatus 22 from the visualfield image to the wide area image, the visual field image may beinstantaneously switched to the wide area image. In addition, the visualfield image may be zoomed out, or the point of view may be moved tothereby gradually change the visual field image to the wide area image.

The visual field image can be switched to the wide area image when, forexample, a user (for example, user A) wants to know the visual field(including the direction of the visual field) of another user (forexample, user B). In addition, even in a case where there are no otherusers, the visual field image can be switched to the wide area imagewhen the user A wants to figure out the position of the visual field ofthe user in the entire content image or the point of view of the user(in a case where the content is a free viewpoint image).

Examples of a trigger for switching the visual field image to the widearea image are as follows.

For example, an operation of a key of a remote control included in theinput unit 58 (FIG. 2 ) operated by the user may be the trigger, and thevisual field image may be switched to display the wide area image onlyin a period in which the user is pressing the key of the remote control.

In addition, a voice command spoken by the user may be the trigger, forexample. The visual field image may be switched to the wide area imagein response to a predetermined voice command “zoom out” spoken by theuser, and the wide area image may be returned to the visual field imagein response to a voice command “return” spoken by the user.

In addition, a motion of the head of the user may be the trigger, forexample. The visual field image may be switched to a bird's eye view orthe like as a wide area image when the user faces right below. Thevisual field image may be switched to an image of a 2D map as a widearea image corresponding to the content space when the user faces rightabove.

In addition, communication between users may be the trigger, forexample. The image may be switched to the wide area image when the userstarts to communicate with another user or when the distance fromanother user during communication has increased so that the other useris out of the visual field. After that, the visual field image and thewide area image may be switched in response to a predetermined gestureoperation, such as a nod of the user or a motion of the hand.

In addition, an event in the content may be the trigger, for example. Inresponse to the event, the visual field image may be switched to thewide area image including the place of the event.

Furthermore, in a case where, for example, the content is a live imagein a real space, generation of an alert in the real space (for example,discovery of suspicious person, fire, operation of emergency button, orthe like) may be the trigger. In response to the alert, the visual fieldimage may be switched to the wide area image including the location ofthe generation of the alert.

Furthermore, the content of a conversation as communication betweenusers may be analyzed, and the switch between the visual field image andthe wide area image may be triggered when the result of analysisindicates predetermined content.

In addition, the visual field image and the wide area image may beswitched in response to a gesture (action) of the user looking to theleft or right or a gesture of pulling.

Next, FIG. 15A, FIG. 15B, FIG. 15C, FIG. 15D, and FIG. 15E are diagramillustrating an example of display of the wide area image switched fromthe visual field image. FIG. 15A is an example of display of a visualfield image 181 displayed on the display apparatus 22 worn by the userA.

FIG. 15B is an example of display of a wide area image 182 using abird's eye view switched from the visual field image 181 (FIG. 15A) inresponse to the detection of the trigger. A visual field mark 183indicating the position (that is, point of view) of the user and thedirection of the visual field is superimposed on the wide area image182. The bird's eye view is displayed at an angle that the visual fieldmark 183 is out of the shadow of objects, such as main buildings, on thewide area image 182, while it is prioritized to make it easy tounderstand the current visual field of the user indicated by the visualfield mark 183.

FIG. 15C is an example of display of a wide area image 184 using anequirectangular projection switched from the visual field image 181(FIG. 15A) in response to the detection of the trigger. Visual fieldrange display 185A representing the visual field of the user A issuperimposed on the wide area image 184.

FIG. 15D is an example of display of a wide area image 186 using anequirectangular projection switched from the visual field image 181(FIG. 15A) triggered when, for example, the user A starts to communicatewith the user B. Visual field range display 187A representing the visualfield of the user A and visual field range display 187B representing thevisual field of the user B are superimposed on the wide area image 186.

FIG. 15E is an example of display of a wide area image 188 using anequirectangular projection switched from the visual field image 181(FIG. 15A) triggered when, for example, the user A starts to communicatewith the user B. The wide area image 188 is an expanded image of therange including the visual fields of the user A and the user B. Visualfield range display 189A representing the visual field of the user A andvisual field range display 189B representing the visual field of theuser B are superimposed on the wide area image 188.

As for the wide area image, when the visual fields of a plurality ofusers significantly vary in the case where the plurality of userscommunicates while viewing the content, an aerial view or the likeincluding the visual fields of the plurality of users (visual fieldimages of the plurality of users) at the same time can be adopted as thewide area image and automatically displayed. As for the content of AR,for example, a separate camera can be used to take a picture of the realworld, and the image can be used to generate the wide area image.

Furthermore, in a case where there are a user wearing AR glasses andanother user remotely viewing the content of the image of a live cameraarranged in the space of the user, an aerial image including the visualfields of the user and the other user at the same time can be adopted asthe wide area image and automatically displayed during the communicationbetween the user and the other user.

Furthermore, in the case where the visual fields of a plurality of userssignificantly vary, the display of the wide area image can beprioritized over the display of the visual field image.

Furthermore, in a case where it is estimated that a plurality of userspays attention to different objects in the content image, an imageincluding the different objects (image in which the different objectscan be viewed) can be adopted as the wide area image.

In addition, the wide area image can be an image, in which the image isreduced to display, at a super wide angle, a predetermined range aroundthe visual field image of the user in the content image, and thepositions of the other users are displayed.

Furthermore, as for the aerial view as the wide area image, the point ofview of the aerial view and the angle of view of the aerial view can beselected based on the direction of the HMD as the display apparatus 22.

The display of the wide area image may be triggered when the userrecognizes another user and starts a conversation, that is, for example,when the user selects the other user from a menu including display ofthe icons of the other users and starts a voice call or a text chat.

In addition, the display of the wide area image may be triggered whenanother user moves away from the user in the middle of a conversation,such as a voice call, between the user and the other user.

Note that in the case where the content is a free viewpoint image, animage including a scene of the content space in the free viewpoint imageviewed in an arbitrary direction from an arbitrary point of view can bethe visual field image, and the visual field image can be displayed onthe HMD as the display apparatus 22. However, the movable range of theuser wearing the HMD is limited. Therefore, as for the movement of thevisual field image, the user may be allowed to move the image in therotation direction, and the user apparatus 20 may automatically performthe parallel translation. In the case of viewing such a free viewpointimage, the wide area image can also be displayed as described above.

In addition, the wide area image can be displayed to cover the entirefield of view of the user. Furthermore, the wide area image can bedisplayed below the visual field of the user so as to be attached to theground when the user faces below. In addition, a window can be displayedin part of the visual field image, and the wide area image can bedisplayed in the window.

Furthermore, the wide area image can be faintly displayed with lowerbrightness than the visual field image.

According to the wide area image, in a case where the visual fields ofthe user and the other user are different so that the communication isnot smooth, the wide area image including the visual fields of the userand the other user can be displayed, and the obstacle (gap) of thecommunication can be improved.

Furthermore, according to the wide area image, the user who cannotrecognize the position in the free viewpoint image can easily return toa desirable position or can make a long-distance movement in a shorttime.

<Image Switching Process of Switching Visual Field Image and Wide AreaImage>

Next, FIG. 16 is a flow chart describing an image switching process ofswitching the visual field image and the wide area image.

However, the movement of the visual field of the user will not bementioned in the description of the image switching process.

In step S11, the information processing apparatus 21 generates thevisual field image corresponding to the visual field of the user andcauses the display unit 73 to display the generated visual field image.

Specifically, the image cutting unit 54 of the information processingapparatus 21 cuts out the visual field image from the content image heldin the content holding unit 52 to generate the visual field image, andthe display control unit 60 supplies the image signal of the generatedvisual field image to the display apparatus 22 to cause the display unit73 to display the visual field image.

In step S12, the information processing apparatus 21 determines whetheror not there is a trigger for instructing a switch between the visualfield image and the wide area image. Specifically, the trigger detectionunit 59 of the information processing apparatus 21 determines whether ornot a wide area trigger for instructing a switch to the wide area imageis detected based on a key operation from the user, a speech of theuser, an image of the user, or the like input by the input unit 58.Here, in a case where the trigger detection unit 59 determines that thewide area trigger is not detected, the process returns to step S11. As aresult, the visual field image is still displayed on the display unit73.

On the other hand, in a case where the trigger detection unit 59determines that the wide area trigger is detected in step S12, theprocess proceeds to step S13.

In step S13, the information processing apparatus 21 generates the widearea image, and the display apparatus 22 switches the display of thedisplay unit 73 from the visual field image to the wide area image.

Specifically, for example, the display control unit 60 superimposes thevisual field range display representing the visual field of each user(for example, visual field range display 187A and 1878 of FIG. 15D) onthe entire image 100 (FIG. 4 ) of the content image held in the contentholding unit 52 to thereby generate the wide area image (for example,wide area image 186 of FIG. 15D) and supplies the image signal of thegenerated wide area image to the display apparatus 22 to cause thedisplay unit 73 to display the wide area image.

Note that the display control unit 60 can determine the display positionof the visual field range display representing the visual field of eachuser based on the visual field information of each user held in thevisual field information holding unit 55.

After step S13, the process proceeds to step S14, and the triggerdetection unit 59 determines whether or not the visual field trigger forinstructing a switch to the visual field image is detected based on akey operation from the user, a speech of the user, an image of the user,or the like input by the input unit 58. Here, in a case where thetrigger detection unit 59 determines that the visual field trigger isnot detected, the process returns to step S13. As a result, the widearea image is still displayed on the display unit 73.

In a case where the trigger detection unit 59 determines that the visualfield trigger is detected in step S14, the process returns to step S11,and the display of the display unit 73 is switched to the visual fieldimage.

According to the image switching process described above, the user canperform an operation, a speech, an action, a gesture, or the like as atrigger to switch the visual field image and the wide area image.

The user viewing the wide area image can figure out the visual fields ofthe other users, and the situation, such as miscommunication, can besuppressed in the communication between the users.

In addition, the user viewing the wide area image can figure out thevisual field and the point of view of the user (position of the user inthe content space in the case where the content is a free viewpointimage). Therefore, in a case where, for example, the user cannotrecognize the position of the user in the content space, the user canquickly return to a desirable position.

<7. Movement of Visual Field Image>

Next, movement of the visual field image will be described.

In the HMD, the user rotates the head to move the visual field image(range of the visual field image) to be cut out from the content image,and as a result, the visual field of the user moves. However, forexample, in a case of moving the visual field image to right behind thevisual field image currently displayed on the display apparatus 22 inthe content image, the body also needs to be rotated along with thehead, because a person usually cannot rotate only the head 180 degrees.Therefore, if the user frequently looks to the left and right or theback while wearing the HMD, the user gets physically tired.

Thus, the user apparatus 20 in the present embodiment moves the visualfield image (visual field center indicated in the visual field centerinformation included in the visual field information for determining thevisual field image) more than the amount of rotation of the head toreduce the physical fatigue of the user. That is, the angle of rotationof the visual field image is made larger than the angle of rotation ofthe head of the user. For example, the visual field image is rotated andmoved 180 degrees in the horizontal direction when the user rotates thehead 90 degrees in the horizontal direction.

FIG. 17A and FIG. 17B are diagrams illustrating an example in which thevisual field image is rotated and moved more than the amount of rotationof the head of the user.

FIG. 17A illustrates a case of viewing, from above, a state in which theuser faces the front in a predetermined direction and views a sphericalimage as a content image 200. In the content image 200, there is anobject 201 in front of the user, and there is an object 202 behind theuser. In this case, the object 201 exists in a visual field image (rangeof visual field image) 203 to be cut out from the content image 200 whenthe user faces the front, and the user can visually recognize the object201. On the other hand, the object 202 exists outside the visual fieldimage 203 of the user, and the user cannot visually recognize the object202.

When the user rotates the head 90 degrees in this state, the visualfield image 203 is rotated, for example, 180 degrees, which is largerthan the angle of rotation of the head, in the same direction as therotation of the head in the content image 200 as illustrated in FIG.17B. In this case, the object 201 exists outside the visual field image203 after the rotation, and the user cannot visually recognize theobject 201. However, the object 202 exists in the visual field image 203after the rotation, and the user can visually recognize the object 202.

However, in a case where the visual field image is simply moved by equalto or more than the amount of rotation of the head as illustrated inFIG. 17A and FIG. 17B, the user may feel, for example, unpleasant orsick. Therefore, in the user apparatus 20, the visual field image can berotated and moved more than the amount of rotation of the head only in acase where the user takes an action which can be assumed that the useris strongly willing to greatly move the visual field or under acondition that the user is unlikely to notice a large movement of thevisual field.

<Movement of Visual Field Image According to Movement of Line of Sight>

The user apparatus 20 can move the visual field image to thereby movethe visual field of the user when only the line of sight is movedwithout the rotation of the head.

FIG. 18A, FIG. 18B, FIG. 19A, and FIG. 19B are diagrams illustrating therelationship between the motion of the line of sight of the user and themovement of the visual field image.

FIG. 18A is an example of an entire image 190 obtained by developing thecontent image on a plane, and a range 191 on the entire image 190indicates a range cut out as a visual field image 192 (FIG. 18B) to bedisplayed on the display unit 73 of the display apparatus 22. FIG. 18Bis an example of display of the visual field image 192, and an X markindicates a line-of-sight position 195 of the user on the visual fieldimage 192. In the case of FIG. 18B, the line-of-sight position 195 ispositioned at the lower center of the visual field image 192.

FIG. 19A indicates a motion of the line of sight of the user. In FIG.19A, the line-of-sight position 195 of the user is moved to the lowerside of the left edge of the visual field image 192. FIG. 19Billustrates a range cut out as the visual field image after the movementof the visual field image according to the movement of the line ofsight, that is, after the movement of the line of sight illustrated inFIG. 19A.

In a case where the user moves the line of sight so that theline-of-sight position 195 approaches the edge of the visual field imageas illustrated in FIG. 19A, it can be assumed that the user is willingto see the outside the range 191 of the current visual field image.Therefore, the visual field image determination unit 53 moves the visualfield image based on the line-of-sight position 195 in response to theapproach of the line-of-sight position 195 to the edge of the visualfield image. That is, the range cut out as the visual field image ismoved from the range 191 (FIG. 17A) to a range 196 (FIG. 18B) in thesame direction as the movement direction of the line of sight. Themovement of the visual field image based on the line-of-sight position195 (of the line of sight after the movement) as described above can bemade by, for example, setting the line-of-sight position 195 as thevisual field center to determine the visual field image to be cut outfrom the content image.

<Movement of Visual Field Image According to Amount of Movement of Lineof Sight and Amount of Rotation of Head>

In a case where the user swiftly moves the line of sight and rotates thehead, it may be difficult for the user to notice a large movement of thevisual field. Therefore, the user apparatus 20 moves the visual fieldimage more than the amount of rotation of the head in a case where theangle of rotation as the amount of movement of the line of sight of theuser and the amount of rotation (angle of rotation) of the head in acertain time are equal to or greater than a threshold.

However, in a case where the rotation direction of the head and themovement direction of the line of sight are different, it is likely thatthe user is viewing the object in the currently displayed visual fieldimage, and the user may feel awkward or unpleasant if the movement ofthe visual field image is equal to or greater than the actual rotationof the head in such a state. Therefore, in a case where the rotationdirection of the head and the movement direction of the line of sightare opposite directions, the movement of the visual field image is madeequal to or greater than the actual rotation of the head only in a casewhere the difference between the rotation of the head and the movementof the line of sight (line-of-sight movement angle described later) ischanged more than a threshold within a certain time.

<Definition of Line-of-Sight Movement Angle and Head Rotation Angle>

FIG. 20 is a diagram for describing definition of a line-of-sightmovement angle representing the amount of movement of the line of sightof the user, and FIG. 20 illustrates a state of viewing, from above, thevisual field of the user provided by the display apparatus 22. Asillustrated in FIG. 20 , an angle from a visual field middlerepresenting the middle of the visual field to a line-of-sight directionafter the movement of the line of sight is defined as a line-of-sightmovement angle representing the angle of rotation that is the amount ofmotion of the line of sight of the user. Furthermore, for example, anangle from the visual field middle to the left edge of the visual fieldis defined as a positive line-of-sight movement angle, and an angle fromthe visual field middle to the right edge of the visual field is definedas a negative line-of-sight movement angle.

FIG. 21A, FIG. 21B, and FIG. 21C are diagrams for describing thedefinition of the head rotation angle representing the amount ofrotation of the head of the user, and FIG. 21A, FIG. 21B, and FIG. 21Cillustrate a situation of viewing, from above, the state in which theuser views a spherical image as the content image 200. The head rotationangle is defined such that a predetermined direction in which the headof the user is currently facing indicated by FIG. 21A is 0 degrees.Furthermore, for example, an angle from the predetermined direction tothe left side is defined as a positive head rotation angle, and an anglefrom the predetermined direction to the right side is defined as anegative head rotation angle.

Note that in FIG. 21A, the head rotation angle is 0 degrees, and theline-of-sight movement angle is a positive value. In FIG. 21B, the headrotation angle is a positive value, and the line-of-sight movement angleis a positive value larger than the line-of-sight movement angle in FIG.21A. In FIG. 21C, the head rotation angle is a positive value, and theline-of-sight movement angle is a negative value.

<Visual Field Image Movement Process According to Amount of Movement ofLine of Sight and Amount of Rotation of Head>

Next, a visual field image movement process according to the amount ofmovement of the line of sight and the amount of rotation of the headwill be described.

FIG. 22 is a flow chart describing a visual field image movement processof moving the visual field image according to the amount of movement ofthe line of sight and the amount of rotation of the head. The visualfield image movement process is executed while the content is viewed.

In step S21, the head motion detection unit 72 of the display apparatus22 (FIG. 3 . Starts to acquire the head rotation angle of the user. Thevisual field image determination unit 53 of the information processingapparatus 21 (FIG. 2 ) is notified of the acquired head rotation angleof the user. In step S22, the line-of-sight detection unit 71 of thedisplay apparatus 22 detects the line of sight of the user and starts toacquire the line-of-sight position 195 (FIG. 18A and FIG. 18B) and theline-of-sight movement angle of the user. The visual field imagedetermination unit 53 of the information processing apparatus 21 isnotified of the line-of-sight position 195 and the line-of-sightmovement angle.

In step S23, the visual field image determination unit 53 determineswhether or not the line of sight of the user has approached the edge ofthe visual field image based on the notification from the line-of-sightdetection unit 71 and advances the process to step S24 when determiningthat the line of sight of the user has approached the edge of the visualfield image. In step S24, the visual field image determination unit 53moves the visual field image in the same direction as the movementdirection of the line of sight from the current position in the contentimage. As a result, the visual field image displayed on the display unit73 is moved in the direction of the line of sight.

Note that in a case where the visual field image determination unit 53does not determine that the line of sight of the user has approached theedge of the visual field image in step S23, step S24 is skipped, and theprocess proceeds to step S25.

In step S25, the visual field image determination unit 53 determineswhether or not a combined angle of the head rotation angle and theline-of-sight movement angle of the user in a certain time is equal toor greater than a threshold. In a case where the visual field imagedetermination unit 53 determines that the combined angle of the headrotation angle and the line-of-sight movement angle of the user in acertain time is equal to or greater than the threshold, that is, forexample, in a case where the head and the line of sight of the user moveat equal to or higher than a certain speed in the same direction or in acase where one of the motions (rotations) of the head and the line ofsight of the user is sufficiently larger than the other although thehead and the line of sight move in opposite directions, the processproceeds to step S26. In step S26, the visual field image determinationunit 53 moves (rotates) the visual field image more than the headrotation angle from the current position in the content image. Note thatan arbitrary method can be used to calculate the amount of movement(angle of rotation) of the visual field image in moving the visual fieldimage more than the head rotation angle. The process then returns tostep S23, and the subsequent process is repeated.

On the other hand, in a case where the visual field image determinationunit 53 determines that the combined angle of the head rotation angleand the line-of-sight movement angle of the user in a certain time issmaller than the threshold in step S25, the process proceeds to stepS27. In step S27, the visual field image determination unit 53 moves thevisual field image according to the head rotation angle from the currentposition in the content image. The process then returns to step S23, andthe subsequent process is repeated.

According to the visual field image movement process described above, itcan be assumed that the user is willing to view the outside (sceneoutside) the current visual field image (scene in the visual fieldimage) in the case where the user moves the line of sight to the edge ofthe visual field image. Therefore, the visual field image is moved inthe direction in which the user has moved the line of sight.

Furthermore, in the case where the combined angle of the head rotationangle and the line-of-sight movement angle of the user in a certain timeis equal to or greater than the threshold, it may be difficult for theuser to notice a large movement of the visual field. Therefore, thevisual field image is moved more than the head rotation angle. As aresult, the user can, for example, rotate the head in some degree tomove the visual field to right behind without rotating the body.Therefore, the visual field can be easily moved in viewing the content,without increasing the physical load.

<8. Selection of Avatar to Be Displayed>

In the case where the content is a free viewpoint image when a pluralityof users views the same content, each user can arbitrarily change thepoint of view of the user. Therefore, another user (point of view ofanother user) may exist in the visual field of a user (visual fieldimage viewed by the user). In such a case, a character as a model of theother user, such as an avatar, can be displayed (superimposed) at theposition of the other user (point of view of the other user) in thevisual field image (content space in the visual field image). The avatarmay be a drawing or a photograph. In addition, the avatar may betwo-dimensionally displayed or three-dimensionally displayed.

The display of the avatar of the other user on the visual field imagecan expect an advantageous effect that the users can more smoothlycommunicate.

However, if there are a plurality of other users in the visual fieldimage, and all of the avatars corresponding to the plurality of otherusers are displayed, there may be a trouble, such as the visual fieldimage is occupied by the avatars of the other users (avatarscorresponding to the other users) or finding the partner (avatar) ofcommunication becomes difficult.

Therefore, in the case of displaying the avatars, a mechanism ofselecting the avatars to be displayed on the visual field image can beimplemented.

In the user apparatus 20 of the present embodiment, the display controlunit 60 (FIG. 2 ) of the information processing apparatus 21 selects theavatars to be displayed (superimposed) on the visual field image.

FIG. 23 is a plan view for describing a method of selecting avatars tobe displayed on the visual field image in the display control unit 60.FIG. 23 illustrates a case of viewing, from above, a content space 301in the case where a plurality of users views the content of the samefree viewpoint image.

The content space 301 includes an avatar 311A corresponding to a user A,an avatar 311B corresponding to a user B, an avatar 311C correspondingto a user C, and an avatar 311D corresponding to a user D. FIG. 23illustrates a range 312A in the visual field image of the user A, arange 312B in the visual field image of the user B, a range 312C in thevisual field image of the user C, and a range 312D in the visual fieldimage of the user D.

The display control unit 60 of the information processing apparatus 21of the user apparatus 20 can set priorities of other users (avatars ofother users), among the other users (avatars) existing in the visualfield image of the user, in descending order of possibility of viewingthe same thing as the user, that is, for example, in descending order ofoverlapping area of the range in the visual field image (here, area asviewed in the plan view of FIG. 23 ), and display the avatars of theother users according to the priorities.

For example, the display control unit 60 of the information processingapparatus 21 of the user apparatus 20 used by the user A specifies theother users (in the case of FIG. 23 , users B, C, and D) existing in therange 312A in the visual field image of the user A and detects theoverlapping areas of the range 312A in the visual field image of theuser A and the ranges in the visual field images of the other users.Specifically, the display control unit 60 detects the area of each of anoverlapping range 315AB of the range 312A and the range 312B in thevisual field image of the user B, an overlapping range 315AC of therange 312A and the range 312C in the visual field image of the user C,and an overlapping range 315AD of the range 312A and the range 312D inthe visual field image of the user D.

In the case of FIG. 23 , the area of the overlapping range 315AD is thelargest, followed by the overlapping range 315AB and the overlappingrange 315AC. Therefore, the priorities of displaying the avatars aredetermined in order of the user D, the user B, and the user C.Furthermore, the avatars are displayed based on the priorities accordingto the number of avatars of the other users displayed in the visualfield image of the user A. Note that the number of avatars of the otherusers displayed in the visual field image are determined in advance, andthe user can change the number of avatars.

FIG. 24 is a diagram corresponding to the state illustrated in FIG. 23 ,the diagram illustrating an example of display of the visual field imageof the user A displayed on the display apparatus 22 of the userapparatus 20 used by the user A and the avatars corresponding to theother users displayed in the visual field image. Here, FIG. 24corresponds to a case where the number of avatars of the other usersdisplayed in the visual field image is one.

As illustrated in FIG. 24 , the avatar 311D corresponding to the user Dindicated by a solid line is superimposed and displayed on the visualfield image 321A of the user A. Note that the avatar 311B correspondingto the user B and the avatar 311 corresponding to the user C indicatedby dotted lines are not displayed. However, in a case where the numberof avatars of the other users displayed in the visual field image istwo, the avatar 311D and the avatar 311B are displayed. In a case wherethe number of avatars of the other users displayed in the visual fieldimage is three or more, the avatars 311D, 311B and 311C are displayed.

Note that here, the priority of the avatar to be displayed in the visualfield image of the user A is set according to the overlapping area ofthe range in the visual field image of the user A and the range in thevisual field image of another user. Therefore, the avatar of anotheruser with the visual field close to the visual field image (range in thevisual field image) of the user A tends to be preferentially displayedin the visual field image of the user A.

Other than setting the priority of displaying the avatar in the visualfield image of the user A according to the proximity of the visual fieldof the other user and the visual field image of the user A as describedabove, the priority can be set according to the positional relationshipbetween the visual field image (range in the visual field image) of theuser A and the visual field of the other user. In this way, whether ornot to display (superimpose) the avatar of the other user on the visualfield image of the user A can be controlled according to the positionalrelationship between the visual field image of the user A and the visualfield of the other user.

<Avatar Display Process>

Next, FIG. 25 is a flow chart describing an avatar display process ofsetting the priorities of the avatars to display the avatars asdescribed above.

In step S31, the display control unit 60 reads the visual fieldinformation of the user and the visual field information of the otherusers held in the visual field information holding unit 55. In step S32,the display control unit 60 refers to the visual field information ofthe user and the other users to determine whether or not the visualfield image of the user, that is, the visual field image (range in thevisual field image) of the user determined according to the visual fieldinformation of the user, includes more than a predetermined number ofother users (points of view of the other users). The predeterminednumber corresponds to the preset number of avatars of the other usersdisplayed in the visual field image of the user.

In step S32, the process proceeds to step S33 in a case where thedisplay control unit 60 determines that the visual field image of theuser includes more than the predetermined number of other users (pointsof view of the other users). In step S33, the display control unit 60refers to the visual field information of the user and the other usersto set the priorities for the avatars of the other users in descendingorder of the size of the range where the visual field (range in thevisual field image) overlaps the visual field image (range in the visualfield image) of the user.

In step S34, the display control unit 60 determines the predeterminednumber of avatars of the other users (avatars corresponding to the otherusers) to be displayed in the visual field image of the user accordingto the set priorities and superimposes the determined avatars of theother users on the visual field image. In this way, the predeterminednumber of avatars are superimposed and displayed on the visual fieldimage displayed on the display unit 73.

On the other hand, in a case where the display control unit 60determines that the visual field image of the user does not include morethan the predetermined number of other users (points of view of theother users) in step S32, all of the avatars of the other users existingin the visual field image of the user are to be displayed, and theprocess proceeds to step S35.

In step S35, the display control unit 60 determines that all of theavatars of the other users existing in the visual field image of theuser are the avatars to be displayed in the visual field image andsuperimposes the determined avatars of the other users on the visualfield image. In this way, equal to or smaller than the predeterminednumber of avatars are superimposed and displayed on the visual fieldimage displayed on the display unit 73.

According to the avatar display process described above, the user cansee only the other avatars considered to be viewing the same or similarvisual field images, and the user cannot see the other avatars. This cansuppress occurrence of a problem in communication, such asmiscommunication with the other users corresponding to the avatars thatcan be viewed. This can also suppress a situation that the visual fieldof the user is occupied by the avatars.

In addition, the user may be able to arbitrarily set the prioritiesregarding the display of the avatars corresponding to the other users.In addition, the priorities may be set according to a history ofcommunication, such as for example, the higher the number of exchangesof messages, the higher the priority. Furthermore, the priorities setbased on these various standards may be able to be switched.

Furthermore, in a case where the content is content that changes withtime, the priorities may be set by considering not only the area of theoverlapping range of the visual field image (range in the visual fieldimage), but also the time of overlap of the visual field image.

Note that the avatar display process can also be applied to cases ofdisplaying an icon of the user (icon corresponding to the user), alive-action image of the user, a character (string) (image of character)representing the user, or other user images representing the user,instead of displaying the avatar.

Furthermore, the icon may be displayed at first, and the display may bechanged to display only the contour of the avatar, monochrome display ofthe avatar, full color display of the avatar, or the like according tothe closeness of the relationship between the users or the degree of thecommunication.

In addition, mutual authentication may also be performed between theusers before the icon or the avatar is displayed.

Note that for the content of either one of the free viewpoint image andthe spherical image, various methods can also be used to control whetheror not to display (superimpose) the user images, such as avatars, of theother users on the visual field image of the user.

For example, the avatar of another user with the visual field close tothe user, that is, another user with a similar visual field image, canbe displayed in the visual field image of the user, at a position closeto the user in the visual field image of the user.

In addition, the avatar of another user who might get along with theuser may be preferentially displayed on the visual field image of theuser according to the proximity of seek positions of the same contentviewed by the user and the other user (temporal positions of reproducedcontent), the proximity of the points of view of the user and the otheruser in the case where the content is a free viewpoint image of a 3Dgame or the like, a history of past conversation between the user andthe other user, the proximity of relationship in an SNS, or the like.

Furthermore, for the avatar of each user, user information regarding theuser such as, for example, the content of conversation (text chat) ofthe user and the nickname of the user, can be displayed along with theavatar. In the case where the avatar of another user can be displayed onthe visual field image of the user, the display regarding the avatar ofthe other user to be displayed in the visual field image of the user canbe limited according to the relationship between the user and the otheruser or the like.

For example, in a case where the avatar (position of the avatar) ofanother user is in the visual field of the user, but the avatar of theuser is not in the visual field of the other user, the details of thedisplay regarding the avatar of the other user, that is, for example,the angle of the neck and the expression of the avatar of the otheruser, the text of the conversation, and the like, may not be displayedin the visual field image of the user, and a simple avatar, that is, forexample, only a silhouette (external form) of the avatar, may bedisplayed. Furthermore, a dialog for asking permission of communicationmay be displayed in the visual field image of each of the user and theother user, and in a case where each of the user and the other userpermits the communication, the display regarding the avatar of thepartner may be displayed in the visual field image. This case canprevent the user from peeping the communication of the other user. Notethat in addition, only the icon of the partner may be displayed beforeeach of the user and the other user permits the communication, and inthe case where each of the user and the other user permits thecommunication, the avatar of the partner or a silhouette of live actionmay be displayed.

Furthermore, in the case of displaying the avatar of the other user inthe visual field image of the user, a 3D image or the like may beadopted as the avatar of the other user. The visual field image may bedisplayed such that the visual field image comes around the avatar ofthe other user according to a motion of the neck or the body of theuser, and this allows the user to feel that the existence of the otheruser is real.

Furthermore, when another user turns the neck to move away in the visualfield image of the user in the case where the avatar of the other userwith the visual field close to the user, that is, the other user with asimilar visual field image, is displayed in the visual field image ofthe user, the avatar of the other user may be moved to a position farfrom the user in the visual field image of the user, or the avatar ofanother user (user different from the other user) with the visual fieldclose to the user may be displayed in the visual field image of the userin place of the avatar of the other user.

In addition, the display regarding the avatar of another user to bedisplayed in the visual field image of the user can be changed accordingto the relationship between the user and the other user. For example,part of information (less information) of the display regarding theavatar of another user may be displayed in the visual field image of theuser who has once talked to the other user and registered the other useras a friend. All of display (more information) regarding the avatar ofanother user may be displayed in the visual field image of the user whois talking to the other user.

Furthermore, in a case where the user can perform a zoom-in or zoom-outoperation to adjust the angle of view of the image (range of the scenein the image) displayed on the display apparatus 22, whether the angleof view of the image displayed on the display apparatus 22 of the userand the angle of view of the image displayed on the display apparatus 22of the other user are similar may also be taken into account to set thepriority of displaying the avatar of the other user in the visual fieldimage of the user.

Furthermore, in a case where the content includes a plurality ofspherical images photographed by a plurality of spherical cameras,whether the user and the other user are viewing the spherical imagephotographed by the same spherical camera or the user and the other userare viewing the spherical images photographed by different sphericalcameras may also be taken into account to set the priority of displayingthe avatar of the other user in the visual field image of the user.

Furthermore, in the case where the content is a free viewpoint image,the proximity of the visual fields of the user and the other user may bedetermined according to, for example, the angles of the necks of theuser and the other user, whether the same substance (object) is in thevisual field images of the user and the other user, and the like.Whether the same substance is in the visual field images of the user andthe other user may be determined by, for example, a combination of thepositions (points of view) of the user and the other user and the anglesof view of the visual field images of the user and the other user.

Furthermore, regarding the display of the avatar, a mode of displayingthe avatar of another specific user and a mode of preferentiallydisplaying the avatar of another user who has talked to the user in thepast may be provided in the visual field image of the user, and the modeof displaying the avatar may be switched to the mode according to theoperation of the user.

In this way, the avatars can be displayed to allow a large number ofunspecified users viewing the same or similar scene in the same contentto connect to each other, and the conversation can be smoothly advanced.Furthermore, the priorities can be set for the avatars (other users)according to the proximity of visual fields, the permission ofcommunication, and the like, and the avatars can be displayed accordingto the priorities. This can prevent a large number of avatars fromcovering the entire visual field image which disturbs viewing thecontent.

<9. A Series of Processes Executed by Software>

The series of processes can be executed by hardware or can be executedby software. In the case where the series of processes are executed bysoftware, a program included in the software is installed on a computer.Here, examples of the computer include a computer incorporated intodedicated hardware and a general-purpose personal computer or the likethat can execute various functions by installing various programs.

FIG. 26 is a block diagram illustrating a configuration example ofhardware of a computer that uses a program to execute the series ofprocesses.

In a computer 500, a CPU (Central Processing Unit) 501, a ROM (Read OnlyMemory) 502, and a RAM (Random Access Memory) 503 are connected to eachother through a bus 504.

An input/output interface 505 is further connected to the bus 504. Aninput unit 506, an output unit 507, a storage unit 508, a communicationunit 509, and a drive 510 are connected to the input/output interface505.

The input unit 506 includes a keyboard, a mouse, a microphone, and thelike. The output unit 507 includes a display, a speaker, and the like.The storage unit 508 includes a hard disk, a non-volatile memory, andthe like. The communication unit 509 includes a network interface andthe like. The drive 510 drives a removable medium 511, such as amagnetic disk, an optical disk, a magneto-optical disk, and asemiconductor memory.

In the computer configured in this way, the CPU 501 loads a programstored in the storage unit 208 on the RAM 203 through the input/outputinterface 205 and the bus 204 and executes the program to execute theseries of processes, for example.

The program executed by the computer (CPU 501) can be provided by, forexample, recording the program in the removable medium 511 as a packagemedium or the like. In addition, the program can be provided through awired or wireless transmission medium, such as a local area network, theInternet, and digital satellite broadcasting.

In the computer 500, the removable medium 511 can be mounted on thedrive 510 to install the program on the storage unit 508 through theinput/output interface 505. In addition, the communication unit 509 canreceive the program through a wired or wireless transmission medium toinstall the program on the storage unit 508. Furthermore, the programcan be installed in advance on the ROM 502 or the storage unit 508.

Note that the program executed by the computer 500 may be a program inwhich the processes are executed in chronological order described in thepresent specification, or the program may be a program for executing theprocesses in parallel or for executing the processes at necessary timingsuch as when the processes are invoked.

Note that the advantageous effects described in the presentspecification are illustrative only, and the advantageous effects arenot limited. There may also be other advantageous effects.

The embodiment of the present technique is not limited to the embodimentdescribed above, and various changes can be made without departing fromthe scope of the present technique.

<1>

An information processing apparatus including:

a content acquisition unit configured to acquire data of content;

an image cutting unit configured to cut out a first visual field imagecorresponding to a visual field of a first user from a content imagebased on the data of the content;

a visual field information acquisition unit configured to acquire visualfield information representing a visual field of a second user viewingthe content image; and

a display control unit configured to control a display apparatus todisplay the first visual field image and configured to control thedisplay apparatus to display the visual field of the second user basedon the visual field information of the second user.

<2>

The information processing apparatus according to <1>, in which

the display control unit is configured to control the display apparatusto display a visual field position instruction image indicating aposition of the visual field of the second user based on the visualfield information of the second user.

<3>

The information processing apparatus according to <2>, in which

the visual field position instruction image includes a wide area imageincluding the first visual field image and a second visual field imagecorresponding to the visual field of the second user.

<4>

The information processing apparatus according to <3>, in which

the display control unit is configured to control the display apparatusto switch the first visual field image and the wide area image inresponse to a predetermined trigger.

<5>

The information processing apparatus according to <4>, in which

the predetermined trigger includes at least one of a key operation, avoice command, a motion of a head, or a gesture operation by the firstuser.

<6>

The information processing apparatus according to any one of <3> to <5>,in which

the wide area image includes at least one of an equirectangularprojection, a Mercator projection, a bird's eye view, an aerial view, ora two-dimensional map.

<7>

The information processing apparatus according to any one of <2> to <6>,in which

the visual field position instruction image includes a symbol imagesuperimposed on the first visual field image, the symbol imageindicating the position of the visual field of the second user.

<8>

The information processing apparatus according to any one of <1> to <7>,further including:

a visual field image determination unit configured to determine thefirst visual field image to be cut out from the content image based onat least one of a movement of a line of sight of the first user or amovement of a head of the first user.

<9>

The information processing apparatus according to <8>, in which thevisual field image determination unit is configured to move the firstvisual field image based on the line of sight of the first user inresponse to approach of the line of sight of the first user to an edgeof the first visual field image.

<10>

The information processing apparatus according to <8> or <9>, in which

the visual field image determination unit is configured to make an angleof rotation of the first visual field image larger than an angle ofrotation of the head of the first user based on an angle of rotation ofthe line of sight of the first user and the angle of rotation of thehead of the first user.

<11>

The information processing apparatus according to any one of <8> to<10>, in which

the visual field image determination unit is configured to determine,based on the visual field information of the second user, an initialposition of the visual field image of the first user corresponding totiming that the first user has substantially started to view thecontent.

<12>

The information processing apparatus according to any one of <8> to<11>, in which

the display apparatus includes a head mounted display, and

the visual field image determination unit is configured to determine thefirst visual field image based on at least one of a movement of the lineof sight of the first user or a movement of the head of the first userassociated with the head mounted display.

<13>

The information processing apparatus according to any one of <1> to<12>, in which

the second user includes a plurality of users,

the display control unit is configured to control the display apparatusto superimpose, on the first visual field image, at least one of aplurality of user images corresponding to the plurality of users, and

the display control unit is configured to control whether or not tosuperimpose, on the first visual field image, each of the plurality ofuser images according to a positional relationship between the firstvisual field image and a visual field of each of the plurality of users.

<14>

The information processing apparatus according to <13>, in which

the display control unit is configured to control the display apparatusto preferentially superimpose, on the first visual field image, a userimage corresponding to a user with the visual field relatively close tothe first visual field image among the plurality of users.

<15>

The information processing apparatus according to <13> or <14>, in which

the display control unit is configured to control the display apparatusto preferentially superimpose, on the first visual field image, part ofthe plurality of user images according to a history of communicationbetween the first user and the plurality of users.

<16>

The information processing apparatus according to any one of <1> to<15>, in which

the content image includes a spherical image or a free viewpoint image.

<17>

The information processing apparatus according to any one of <1> to<16>, further including:

the display apparatus.

<18>

An information processing method including:

acquiring data of content;

cutting out a first visual field image corresponding to a visual fieldof a first user from a content image based on the data of the content;

acquiring visual field information representing a visual field of asecond user viewing the content image; and

controlling a display apparatus to display the first visual field imageand controlling the display apparatus to display the visual field of thesecond user based on the visual field information of the second user.

<19>

A program for causing a computer to function as:

a content acquisition unit configured to acquire data of content;

an image cutting unit configured to cut out a first visual field imagecorresponding to a visual field of a first user from a content imagebased on the data of the content;

a visual field information acquisition unit configured to acquire visualfield information representing a visual field of a second user viewingthe content image; and

a display control unit configured to control a display apparatus todisplay the first visual field image and configured to control thedisplay apparatus to display the visual field of the second user basedon the visual field information of the second user.

REFERENCE SIGNS LIST

10 Content viewing system, 20 User apparatus, 21 Information processingapparatus, 22 Display apparatus, 31 Internet, 40 Server apparatus, 41Content distribution unit, 42 Visual field information management unit,43 Communication management unit, 51 Communication unit, 52 Contentholding unit, 53 Visual field image determination unit, 54 Image cuttingunit, 55 Visual field information holding unit, 58 Input unit, 59Trigger detection unit, 60 Display control unit, 71 Line-of-sightdetection unit, 72 Head motion detection unit, 73 Display unit, 74 Voiceinput/output unit, 100 Entire image, 101 Visual field range, 111 Visualfield image, 112 Wide area image, 113 Visual field range display, 121Visual field direction instruction mark, 131 Visual field directioninstruction line, 132 Icon, 141 Visual field range display, 151 Trackingimage, 162 Icon, 163 Arrow, 200 Content image, 301 Content space, 311Avatar, 500 Computer, 501 CPU

The invention claimed is:
 1. An information processing apparatus,comprising: circuitry configured to: acquire first visual fieldinformation that represents a first visual field of a first user in awide area image; acquire second visual field information that representsa second visual field of a second user in the wide area image; andcontrol a display apparatus to: display a first visual field image tothe first user based on the first visual field information; display,based on a detection of a specific trigger, the wide area image to thefirst user; display, based on the display of the first visual fieldimage, a visual field instruction image, wherein the visual fieldinstruction image indicates a position of the second visual field on thefirst visual field image; and display, based on the display of the widearea image, a visual field range, wherein the visual field rangeindicates a range of the second visual field on the wide area image. 2.The information processing apparatus according to claim 1, wherein thevisual field instruction image includes a symbol image superimposed onthe first visual field image, and the symbol image indicates theposition of the second visual field of the second user.
 3. Theinformation processing apparatus according to claim 1, wherein thecircuitry is further configured to control the display apparatus toconcurrently display the first visual field image and the wide areaimage to the first user.
 4. The information processing apparatusaccording to claim 1, wherein the specific trigger includes at least oneof a key operation, a voice command, a motion of a head, or a gestureoperation by the first user.
 5. The information processing apparatusaccording to claim 1, wherein the wide area image includes at least oneof an equirectangular projection, a Mercator projection, a bird's eyeview, an aerial view, or a two-dimensional map.
 6. The informationprocessing apparatus according to claim 1, wherein the circuitry isfurther configured to determine the first visual field image to be cutout from a content image, and the first visual field image is cut outfrom the content image based on at least one of a movement of a line ofsight of the first user or a movement of a head of the first user. 7.The information processing apparatus according to claim 6, wherein amovement of the first visual field image is based on the line of sightof the first user and approach of the line of sight of the first user toan edge of the first visual field image.
 8. The information processingapparatus according to claim 6, wherein an angle of rotation of thefirst visual field image is larger than an angle of rotation of the headof the first user based on an angle of rotation of the line of sight ofthe first user and the angle of rotation of the head of the first user.9. The information processing apparatus according to claim 6, whereinthe display apparatus includes a head mounted display, and the circuitryis further configured to determine the first visual field image based onat least one of the movement of the line of sight of the first user orthe movement of the head of the first user associated with the headmounted display.
 10. The information processing apparatus according toclaim 6, wherein the content image includes one of a spherical image ora free viewpoint image.
 11. The information processing apparatusaccording to claim 1, wherein the second user corresponds to a pluralityof users, and the circuitry is further configured to: control thedisplay apparatus to superimpose, on the first visual field image, atleast one of a plurality of user images corresponding to the pluralityof users; and control superimposition of each of the plurality of userimages on the first visual field image based on a positionalrelationship between the first visual field image and a visual field ofeach of the plurality of users.
 12. The information processing apparatusaccording to claim 11, wherein the circuitry is further configured tocontrol the display apparatus to superimpose, on the first visual fieldimage, a specific user image of the plurality of user images.
 13. Theinformation processing apparatus according to claim 12, wherein thevisual field of the specific user image is relatively close to the firstvisual field image among the plurality of user images.
 14. Theinformation processing apparatus according to claim 11, wherein thecircuitry is further configured to control the display apparatus tosuperimpose, on the first visual field image, part of the plurality ofuser images based on a history of communication between the first userand the plurality of users.
 15. An information processing method,comprising: acquiring first visual field information that represents afirst visual field of a first user in a wide area image; acquiringsecond visual field information that represents a second visual field ofa second user in the wide area image; and controlling a displayapparatus to: display a first visual field image to the first user basedon the first visual field information; display, based on a detection ofa specific trigger, the wide area image to the first user; display,based on the display of the first visual field image, a visual fieldinstruction image, wherein the visual field instruction image indicatesa position of the second visual field on the first visual field image;and display, based on the display of the wide area image, a visual fieldrange, wherein the visual field range indicates a range of the secondvisual field on the wide area image.
 16. A non-transitorycomputer-readable medium having stored thereon computer-executableinstructions which, when executed by a computer, cause the computer toexecute operations, the operations comprising: acquiring first visualfield information that represents a first visual field of a first userin a wide area image; acquiring second visual field information thatrepresents a second visual field of a second user in the wide areaimage; and controlling a display apparatus to: display a first visualfield image to the first user based on the first visual fieldinformation; display, based on a detection of a specific trigger, thewide area image to the first user; display, based on the display of thefirst visual field image, a visual field instruction image, wherein thevisual field instruction image indicates a position of the second visualfield on the first visual field image; and display, based on the displayof the wide area image, a visual field range, wherein the visual fieldrange indicates a range of the second visual field on the wide areaimage.